Skip to content

Pass the TestKit coverage agent on the command line, move the shared TestKit directory - #207

Open
alexander-yevsyukov wants to merge 16 commits into
masterfrom
claude/shared-testkit-home
Open

alexander-yevsyukov wants to merge 16 commits into
masterfrom
claude/shared-testkit-home

Conversation

@alexander-yevsyukov

@alexander-yevsyukov alexander-yevsyukov commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Two problems with the TestKit directory, the Gradle user home of the builds that plugin-testlib runs through Gradle TestKit:

  • Coverage tied it to one module. The TestKit coverage harness attached the JaCoCo agent by writing org.gradle.jvmargs into a gradle.properties in the TestKit directory, so it had to choose that directory itself, build/testkit-home per module. On CI it is empty on every run, so the first TestKit builds of each module redo the work a Gradle user home keeps, chiefly Gradle's instrumented copies of the JARs on the classpath of the plugin under test.
  • The shared directory lived in each checkout. withSharedTestKitDirectory() put it under the repository root, in .gradle-test-kit. In core-jvm-compiler, that directory takes 1.76 GB in local tests, and every new worktree, of the same repository or another one, fills its own copy from scratch. That is costly when several agent sessions work in different worktrees or repositories on one machine.

This PR:

  • Moves the agent to the command line. withTestKitCoverage() appends -Dorg.gradle.jvmargs=-javaagent:… to the runner arguments for builds started by executeTask, executeAndFail and runGradleBuild. The harness no longer chooses a TestKit directory, so coverage and the directory are independent of each other.
  • Changes withSharedTestKitDirectory(). It takes an optional dir: File?. A given directory is used as is. Given none, the project uses caches/spine-testkit under the Gradle user home. The new GradleUserHome.kt looks the Gradle user home up in the order Gradle uses: the gradle.user.home system property, then the GRADLE_USER_HOME environment variable, then .gradle in the user's home directory. It composes the path with File, so Windows paths such as D:\a\.gradle work. @JvmOverloads keeps the method without arguments.
  • Removes RootProject.testKitTempDir(), the old <repository root>/.gradle-test-kit. This is the breaking change.
  • Makes the coverage property names public. TestKitCoverage.AGENT_PROPERTY and EXEC_DIR_PROPERTY hold them, so the spec and the KDoc refer to the constants instead of repeating the strings. config's buildSrc keeps its own copies, because it does not depend on this library.
  • Collects TestKit coverage in plugin-testlib too. Its own tests now run with the agent like the other four TestKit modules. Today this adds no coverage, because the fixtures of GradleProjectSpec run only the java plugin, but it keeps every TestKit module on the same setup.
  • Pulls config, with Say what a shared TestKit home saves config#833 and Set the TestKit directory apart from TestKit coverage config#834. useSharedTestKitDir() now sets Gradle's org.gradle.testkit.dir to caches/spine-testkit on every Test task, apart from enableTestKitCoverage(), so the other TestKit builds of all modules share the directory too. The pull also updates the versions pinned for Spine's own libraries.
  • Adds tests: TestKitCoverageSpec for the appended argument and the no-op cases, GradleUserHomeSpec for the lookup, with two tests that run only on Windows, and new cases in GradleProjectSetupSpec for the choice of the directory.
  • Updates the GradleProject KDoc: the consumer recipe for coverage, and a separate section on the TestKit directory.
  • Bumps the version to 2.0.0-SNAPSHOT.460, as a breaking change, and updates the dependency reports.

An earlier revision added an io.spine.tools.gradle.testkit.coverage.home property for a shared directory, which SpineEventEngine/config#831 set. It duplicated Gradle's org.gradle.testkit.dir and was removed, here and in config#834.

Compatibility

  • Breaking: RootProject.testKitTempDir() is removed.
  • Callers of withSharedTestKitDirectory() still compile and link. They now share caches/spine-testkit under the Gradle user home instead of keeping .gradle-test-kit in each checkout. In the org, these are compiler (4 specs), core-jvm-compiler (4 integration tests) and ProtoTap (1 spec).
  • A Gradle user home given only with -g: the test JVM does not see it, so the default falls back to GRADLE_USER_HOME or ~/.gradle. That only loses the sharing with that build's caches.
  • Older plugin-testlib versions call withTestKitDir(<module>/build/testkit-home) themselves when coverage is on, which overrides org.gradle.testkit.dir, so they keep their per-module directories.
  • Callers that use runner directly: compiler sets the environment and then calls executeTask, so it still gets the agent. ProtoTap and core-jvm-compiler call withJvmArguments(...), and explicit JVM arguments win over org.gradle.jvmargs from any source. Those tests got no worker coverage before this change either.
  • Precedence: the appended argument replaces the test project's org.gradle.jvmargs, as the user-home gradle.properties did before.
  • Build cache: with TestKit coverage on, plugin-testlib's test task is no longer cacheable, like those of the other four modules, so it reruns after clean.
  • License headers: the touched files picked up the repo's current CodeMatters header, which the copyright tooling applies on edit.

Measurements

  • Size: core-jvm-compiler's .gradle-test-kit takes 1.76 GB in local tests. In tool-base, the caches/ part of the shared directory takes about 250 MB, where per-module directories added up to 348 MB.

  • Local wall-clock of the tests of the four coverage modules, measured with the earlier revision:

    TestKit directory Time
    Per module, empty (every CI run before) 37 s
    Shared, empty (one-time) 53 s
    Shared, warm 14–16 s
  • Windows PR run 41670107177, with the shared directory still empty: the build step took 733 s, against 837–901 s on the earlier runs that started with no cache, and no cache-lock timeouts.

  • Coverage: the root Kover report with the TestKit exec files covers 3035 of 3157 lines and 989 of 1188 branches, against 2943 and 970 without them. Codecov reported 92.08% for 4c7ec60, against 92.03% on master, with misses unchanged at 122.

Test plan

  • ./gradlew build dokkaGenerate, locally, after deleting the repository-root .gradle-test-kit and every per-module TestKit directory. None came back, and the TestKit builds wrote into ~/.gradle/caches/spine-testkit.
  • GradleUserHomeSpec: 5 tests pass on macOS. The 2 Windows-only tests are skipped there.
  • The Windows CI job runs the 2 Windows-only tests of GradleUserHomeSpec.
  • GradleProjectSetupSpec, GradleProjectSpec, TestKitCoverageSpec and RootProjectSpec pass.
  • Re-ran the TestKit modules' tests after deleting every build/jacoco-testkit: each of the four coverage modules wrote its testkit.exec (0.9–1.1 MB).
  • With plugin-testlib's coverage on, no class in the root or the module Kover report changed.
  • Pre-PR checks on an earlier revision: spine-code-review, kotlin-engineer, review-docs. The latest changes, the Gradle user home lookup and the new withSharedTestKitDirectory(), have not been re-reviewed.
  • After this PR merges: a master run saves caches/spine-testkit, and later PR runs restore it.

🤖 Generated with Claude Code

alexander-yevsyukov and others added 3 commits November 3, 2026 02:20
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…estKit home

The harness used to attach the JaCoCo agent by writing `org.gradle.jvmargs`
into a `gradle.properties` of a per-module TestKit directory. That tied the
directory to one module, and on CI it was empty on every run, so the first
worker build of each module downloaded its dependencies and generated the
Gradle API JARs again.

The agent now goes into the runner arguments as `-Dorg.gradle.jvmargs`, and
the consuming build may name a TestKit directory shared by all modules via
the optional `io.spine.tools.gradle.testkit.coverage.home` system property.
Without it, each module keeps a directory of its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T02:38:03.161573Z 8549704 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8549704c0e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@codecov

codecov Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.12%. Comparing base (6354e96) to head (a018b5a).

Additional details and impacted files
@@             Coverage Diff              @@
##             master     #207      +/-   ##
============================================
+ Coverage     92.03%   92.12%   +0.09%     
- Complexity      846      848       +2     
============================================
  Files           160      161       +1     
  Lines          3164     3175      +11     
  Branches        381      384       +3     
============================================
+ Hits           2912     2925      +13     
  Misses          122      122              
+ Partials        130      128       -2     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@alexander-yevsyukov alexander-yevsyukov self-assigned this Nov 3, 2026
Base automatically changed from claude/ci-workflow-optimization-ffc95e to master November 3, 2026 15:34
@alexander-yevsyukov alexander-yevsyukov moved this to 🏗 In progress in v2.0 Nov 3, 2026
Copilot AI balanced review requested due to automatic review settings November 3, 2026 16:02
Copilot AI previously approved these changes Nov 3, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

No active review issues are reported for the TestKit coverage harness changes.

No open issues 🎉

What changed in this PR

Updates plugin-testlib’s TestKit coverage harness to pass the JaCoCo agent through runner arguments, allowing worker builds to share a cacheable TestKit home without module-specific configuration files.

Changes:

  • Attaches coverage through build helper arguments and adds optional shared-home selection while preserving caller precedence and the per-module fallback.
  • Adds tests for argument generation, disabled coverage, and directory selection; updates consumer documentation.
  • Bumps the snapshot version, refreshes dependency reports, and incorporates shared-home build configuration.
File Description
version.gradle.kts Bumps the version to 2.0.0-SNAPSHOT.452.
plugin-testlib/​src/​test/​kotlin/​io/​spine/​tools/​gradle/​testing/​TestKitCoverageSpec.kt Tests coverage arguments and TestKit directory selection.
plugin-testlib/​src/​main/​kotlin/​io/​spine/​tools/​gradle/​testing/​TestKitCoverage.kt Moves agent configuration to arguments and supports a shared home.
plugin-testlib/​src/​main/​kotlin/​io/​spine/​tools/​gradle/​testing/​GradleProject.kt Applies coverage arguments and documents setup and direct-runner behavior.
plugin-testlib/​src/​main/​kotlin/​io/​spine/​tools/​gradle/​testing/​Gradle.kt Applies coverage arguments in runGradleBuild.
docs/​dependencies/​pom.xml Updates the reported project version.
docs/​dependencies/​dependencies.md Refreshes report versions and generation timestamps.
buildSrc/​src/​main/​kotlin/​io/​spine/​gradle/​testing/​TestKitCoverage.kt Supplies the shared TestKit home system property.
.github/​workflows/​publish.yml Updates the shared publishing workflow revision.

Review effort: Balanced

alexander-yevsyukov and others added 4 commits November 3, 2026 16:58
Move the three system-property names into `public object TestKitCoverage`,
so that the spec and the `GradleProject` KDoc refer to them instead of
repeating the strings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker builds generate no Gradle API JARs and download few files. Their
Gradle user home mostly holds the instrumented JARs of the plugin classpath.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove `TESTKIT_HOME_PROPERTY` and the per-module `testkit-home` fallback.
With the agent on the command line, the coverage harness needs no TestKit
directory of its own. TestKit takes the one named by Gradle's
`org.gradle.testkit.dir` system property, or a folder under the `Test` task's
temporary directory, and setting that property is up to the consuming build,
apart from enabling coverage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alexander-yevsyukov alexander-yevsyukov changed the title Pass the TestKit coverage agent on the command line, allow a shared TestKit home Pass the TestKit coverage agent on the command line Nov 3, 2026
alexander-yevsyukov and others added 5 commits November 3, 2026 21:30
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The file also picks up the current license header.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`withSharedTestKitDirectory()` now takes an optional `dir`. Given none, a
project uses `caches/spine-testkit` under the Gradle user home, which
`GradleUserHome.kt` looks up as Gradle does: the `gradle.user.home` system
property, then `GRADLE_USER_HOME`, then `.gradle` in the user's home.

Remove `RootProject.testKitTempDir()`, which put the directory under the
repository root.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alexander-yevsyukov alexander-yevsyukov changed the title Pass the TestKit coverage agent on the command line Pass the TestKit coverage agent on the command line, move the shared TestKit directory Nov 3, 2026
Drop the default arguments of `gradleUserHome()`, which nothing used, make
the explicitly given TestKit directory private, and test the error for an
unknown `user.home`. Codecov counted these as uncovered lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alexander-yevsyukov
alexander-yevsyukov requested a balanced review from Copilot November 3, 2026 22:23
Copilot AI dismissed their stale review, a newer Copilot review was requested November 3, 2026 22:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

No active review issues are reported for the changes separating TestKit coverage from directory selection.

No open issues 🎉

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 🏗 In progress

Development

Successfully merging this pull request may close these issues.

2 participants